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SUMMARY 


Subjective estimates of the throat posed by a siOEle Intruder aircraft were 
determined by showing pilots photographs of a cockpit display of traffic information. 
The time the intruder was v'lway from the point of minimum separation was found to be 
the major determinant of the perception of threat. When asked to choose a maneuver 
to reduce the conflict, pilot., selected maneuvers with a bias toward those that wo\.ad 
have kept the intruders in sipjit had they been visible out the cockpit window.. 


INTRODUCTION 


Contemporary advances in ground and cockpit based avionics have made possible 
the placement of electronic u„>,plays of air traffic Information in aircraft cockpits 
(Wempe, 1974). Tliese advances, accordingly raise many questions as to how pilots 
will use this type of information and how their use \d.l], affect existing air traffic 
control procedures (Vorstynen, 1980). 

The traffic information may, for example, allow pilots to assume some responsl" 
bility for maintaining safe spacing from other aircraft under conditions which pres- 
ently would require assistance from ATG (Connelly, 1977; Chappell and PalViier, 1981). 
Uses such as this assume that the information displayed to pilots will enable them 
to detect incipient threatening situations and to take actions to prevent potential 
threats from becoming truly dangerous situations. This assumption, however, must be 
tested since it is not known, g priori , that pilots will be able to detect potential 
threats in a timely manner with acceptable workload levels and respond with appro- 
priate actions. 

Thus, it is important to try to determine what may be called the pilots' "sub- 
jective threat avoidance logic" when viewing a CDTI, in order to help assess what 
new procedures, if any, the installation of such equipment may enable. It is, for 
example, important to assess whether the pilot's "threat logic" Is compatible with 
that of the ground controllers or that of automatic collision avoidance systems. 

Previous experiments concerning pilots' ability to use CDTI to determine the 
future relative position of intruding aircraft have suggested that this task may be 
quite difficult (Stark and Ellis, 1981; Palmer, Jago, and Bnty, 1980), Accordingly, 
the use of CDTI will probably require automatic assistance of the sort provided by 
Automatic Traffic Advisory Resolution Service (ATARS)-type conflict alerts (Lentz 
et al., 1980). The Interaction of automatic conflict alerts with CDTI, however, pro- 
vides another potential problem since the actions suggested by the alerts may con- 
flict with those chosen by a pilot viewing the CDTI. Thus, the presence of CDTI in 
the cockpit may substantially affect the pilot's reaction time to the alerts. 


Another consideration for the introduction of CDTI-type information augmented 
by automatic alerting systems regards pilot acceptance of such an instrument. An 
instrument which frequently loads to incorrect or unnecessary pilot perceptions of 
traffic conflicts can hardly be expected to win pilot approval. Training and exper 
ience, of course, may lead to eventual acceptance; however, the training necessary 
is undoubtedly related to the bias pilots bring to its use. Furthermore, in order 
to insure that Its use will bo predictable and not break down under conditions of 
stress or high workload, the design of the decision logic and display characteris- 
tics of the system should conform, as much as possible, to the preexisting biases 
users bring to the system. 

The most realistic assessment of the potential use of CDTI is, of course, 
through full mission simulation and flight testing. These research techniques are, 
however, costly and preclude the systematic examination of the experimental condi- 
tions generated by even relatively small factorial experimental designs. Accord- 
ingly, part-task experimental designs are required to help select display systems 
for more realistic experiments in simulators and aircraft. The following study is 
one of a series of part-task experiments Uuslgned to run in parallel with part and 
full mission simulations currently examining related questions regarding the use of 
COTI (Chappell and Palmer, 1981). 


METHOD 


Generation of the Display 

Generic CDTI displays were produced using an Evans and Southerland calligraphic 
Picture System controlled by a PDP-11/40 computer. The general format of the dis- 
play was derived from that used in previous experiments (Palmer et al. , 1980; Hart 
and Loomis, 1980). A photograph of two of the displays is shown in figure 1 which 
depicts ovmship on nominal approaches to runway 26'L at Denver Stapleton International 
Airport. Gwnship position is represented as the tip of the chevron two-thirds of 
the way down from the top of the display. As in previous experiments (Palmer et al. , 
1980), the aircraft symbol is augmented by a straight, solid 32 sec predictor and 
dotted line showing 32 sec of history. Ownship barometric altitude in hundreds of 
feet and ground speed in knots were shown in the upper left of the display. The 
current map range from ownship to the top of the map, shown as 30 n. ml. , and the 
mode of the display, heading up, were shown in the upper right corner. The distance 
from ownship position to the airport was approximately 30 n. mi. A single intruding 
aircraft was represented at ownship 's altitude as the apex of a triangle on each 
display and augmented with a similar 32 sec predictor and history. The intruder's 
data tag included the identification "UFO" and two numbers, the left representing 
barometric altitude in hundreds of feet and the right representing true airspeed in 
knots. These above characteristics and the background map remained constant for all 
the static displays that were generated. The choice of map range was based on dis- 
cussions with several pilots who normally fly into Denver along this approach route. 

A four factor central composite design (Clark and Williges, 1973) was used to 
examine the effects of varying the intruder's minimum separation (0, ±0.5, 
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(b) Mean subjective threat (rank) “ 7.7, mean subjective threat (rating) = 4.4, 
minimum horizontal separation = 1.0 n. mi., time to minimum separation ■ 80 sec. 


Figure 1.- Sample traffic conflicts with low perceived threat. 
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tl.O n. mi.), Lime to minimum sepntation (20, 40, 60, 80 sec),* relative velocity 
(0, ±75, ±150 knots), and relative bonding (±30, ±60, ±90“). Once the complete set 
of encounters were generated, they were displayed on the Picture System scopes and 
photographed (see fig. 1) so that high contrast 8xl0 prints could be prepared. Photo- 
graphs were used for the experiment, because previous results suggested that neither 
update rate nor total time viewing developing conflict affects estimates of future 
intruding aircraft position (Palmer et al., 1980). These findings are consistent 
with the observation that a GDTl wliich is updated every 4 sec is an essentially 
static display. 


Procedure 

The experiment was conducted in a small conference room with a table on which 
the photographs were placed. The pilot opinions regarding the encounters were col- 
lected after pilots read a small briefing book (sec appendix) explaining the display 
symbology, scenario assumptions, and threat rating technique. Two different tech- 
niques were used. In method A, the pilots sorted a randomly shuffled stack of 25 
photographs as to the degree of collision threat caused by the Intruder. No specific 
sorting technique was recommended, but all the subjects independently chose first to 
divide the stack into several smaller stacks and then to sort the smaller stacks. In 
method B, the pilots were given another stack of 25 photographs which were mirror 
images of those in the first stack. For each photograph they were to rate the threat: 
of collision in terms of what action they felt they vrould have to take in the 
depicted situation. The two traffic situations illustrated in fig. 1 show sample 
mean threat ranking and rating. The rating scale was expressed in terms of the rela- 
tive urgency of an avoidance maneuver; for example, a rating of 0 indicated the 
intruder could be ignored, a rating of 4 indicated a maneuver needed within 30 sec, 
and a rating of 10 indicated a maneuver was overdue (see appendix). After the pilots 
selected a rating, they chose a maneuver that they thought appropriate from a matrix 
of nine options, including that of no maneuver: 


climb left 

climb 

climb right 

left 

none 

right 

descend left 

descend 

descend right 


Pilots were run in pairs so that while one was using method A the other was 
using method B; after completing the first rating technique the pilots switched 
methods. Thus, the order of rating techniques was counterbalanced across subjects, 
and all pilots used both techniques. Discussion of the encounters was limited to 
questions before and after the actual rating and sorting took place. After complet- 
ing the two evaluation techniques, the pilots were debriefed regarding their previous 
experience with CDTI experiments at Aiies and their experience with flights to Denver. 


^A fifth value of 100 sec for time to minimum separation was accidently omitted. 
Its omission only affects two of the 25 traffic conditions presented to each subject 
and reduces the statistical power of tests on the miss distance variable and all of 
its Interactions. Since no tests of interactions were pi.anned, this omission has a 
negligible effect on the interpretation of the results. 
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Subjecta 


Ten line-qualified airline pilota participated in the experiment. Slwce all 
worked fot major air carriers flying out of Denver, all had substantial experience 
flying there. Seven had previous experience with CDTI experiments, though none had 
to make threat ratings previously. 


RESULTS 


The subjective threat data were collected in two different ways to allow com- 
parison of different dependent measures. The dependent variables used were; the 
rank of the threat from method A and the rating of the threat from method B. These 
variables were each checked for intersubject agreement by calculating Kendall's W 
with correction for ties (Siegel, 1956). All three showed highly statistically sig- 
nificant Intersubject agreement; 

Rank from sorting; W ■ 0.571 , p. < 0.001 

Rating; W « 0,470 , p, < 0,001^ 

However, the values of W show that the pilots were less than unanimous in their 
threat perceptions of the different encounters. Thus, the remaining analyses were 
conducted on a subject-by-subject basis, through some graphs show pooled data. 

Multiple regressions testing only for linear main effects were calculated sep- 
arately for each subject in the manner of central composite designs (Clark and 
Williges, 1973; also see Curry, 1977). In addition to the independent variables of 
minimum separation, time to minimum separation, relative speed, and heading, the 
algebraic sign of minimum separation and headiitg were coded as independent variables 
having values of ±1. In order to separate these two "sign variables" from the parent 
variables from which they were derived, only the absolute value of the parent vari- 
ables, minimum separation and heading, were used for the regression equation. 

These "sign variables" were intended to help determine whether the direction of 
the parent variables rather than their magnitude had an effect on subjective threat. 

A positive minimum separation corresponded to passage of the intruder in front of 
ownship while a negative value corresponded to passage behind. If all aircraft pass- 
ing in front were perceived, for example, as more threatening than aircraft passing 
behind, and this perception were unrelated to the absolute magnitude of miss distance, 
then the sign of minimum separation would contribute to the prediction of subjective 
threat while the absolute value of minimum separation would not. An analogous argu- 
ment can be made with respect to heading and its sign since positive values of head- 
ing indicate intruding aircraft coming from the left and negative values indicate 
aircraft from the right. 

Thus, the multiple regression equations used to predict the subjective threat 
ranking and threat rating for each subject were: 


^For data with no ties the W is proportional to the mean of all the pairwise 
Spearman rank order correlations between the variables. A value of 1.0 represents 
perfect ordinal agreement aaiong the subjects, 0 represents maximum disagreement. 
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T(rank) ■ al(sras) + a2(sh) + a3(mB) + a4(tms) + a5(h) + a6(rs) + aO 

T(rate) " al(sms) + a2(sh) + a3(ma) + a4(tms) + aS,(h) + a6(re) + aO 

« 

where c(rank) and t(rate) are subjective threat estimated by sorting or rating, 
respectively, sms is sign of mininuun separation, sh is sign of heading, ms is 
minimum separation, tros is time to minimum separation, h is heading, and rs is 
relative speed. 

As tables 1 and 2 show, the independent variable showing the most repeated sta- 
tistical significance and a consistent direction of effect across subjects was time 
to minimum separation, which in our encounter situation is similar to horizontal tau 
(range divided by range rate). The negative value of the regression coefficient for 
time to minimum separation indicates that subjective threat decreases as time to 
minimum separation increases. 


TABLE 1.- RANKING FROM SORTING FOR SUBJECTIVE THREAT 


Subj . 

Mult . 
R 

Stand, 
err. of 
reg. 

Coefficients of regression variables 

Const. 

of 

reg. 

Sign 
of min. 
separation 

[ 

Sign 

of 

head- 

ing 

Min, 

separa- 

tion 

Time 
to rain, 
separation 

Head- 

ing 

Rela- 

tive 

speed 

1 

0.90 

3.66 

ns 

ns 

-15.03^ 

-0.24° 

-0., 

12° 

ns 

1.92 

2 

.77 

5.42 





-11.47° 

ns 

-.ir 



3.75 

3 

.93 

3.11 





ns 

-.31° 

ns 



5.33 

4 

.90 

2.66 







-‘K 





12.70 

5 

.75 

5.58 







-.20° 





10.61 

6 

,57 

6.94 







ns 





.02 

7 

.94 

2.93 







-.29° 




1 

10.44 

8 

.94 

2.93 







-.28° 



-9.04^ 

14.46 

9 

.93 

3.09 







-.29° 



ns 

4.92 

10 

.71 

5.98 


’ 


r 


< 

-.38° 



ns 

-.71 


ns: not significant 

%. < 0.05 
^p. < 0.025 

®p. < 0.01 


This relationship between time to minimum separation and subjective threat is ■ 
shown in figure 2 where the mean values of the dependent variables are plotted show- 
ing a strong linear trend. The reliability of the effect shown in this figure was 
checked with Friedman analyses of variance (Siegel, 1956) for each of the methods of 
assessing subjective threat. 

Threat from sorting: chi-square(r) =23.88, df =3, p < 0.001 
Threat from rating: chi-square(r) » 16.44, df = 3, p < 0.001 

The other terms of the regression show only spotty and inconsistent effects 
across subjects. For example, a negative coefficie.'.t for the minimum separation term 














TABLE 2,- RATING OF SUBJECTIVE THREAT 



ns; not significant 

%. < 0.05 
^p. < 0.025 
< 0.01 


^ RANK 



TIME TO MINIMUM SEPARATION, sec 

Figure 2.- Perceived threat as a function of time to minimum separation. 
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indicates that intruders passing farther away were perceived as less threatening. 

This coefficient is negative and statistically significant for oitly one or two sub- 
jects, but the specific subjects showing statistical significance depend upon which 
method of assessing subjective threat is used, Siral3.urly, though a negative coeffi- 
cient for the heading term would Indicate that a more head-on intruder would seem 
less threatening than one coming more from the aide or from behind, this coefficient 
reached statistical significance for only two subjects. Furthermore, the direction 
of effect was inconsistent since the sign of the coefficient was both positive and 
negative depending upon the subject. 

The pilots' selections of maneuvers were analyzed with respect to the matrix of 
options described in the briefing book with the exception that all turning maneuvers 
were mapped into categories of turning toward or away from the intruder. Collapsing 
all subjects together produces table 3, which shows that pilots have a tendency to 
turn toward or descend with respect to the intruder, but that there is no ‘evidence 
of Interaction; that is, the number of composite maneuvers, such as descending turns, 
is not disproportionately large. 


TABLE 3.- BREAKDOWN OF NUMBERS OF CHOSEN MANEUVERS 
(collapsed across subjects) 



Toward 


Away 

Totals 

Climbing 

20 

climbing toward 

25 

climbing 

8 

climbing away 

53 


29 

toward 

20 

none 

11 

away 

60 

Descending 

57 

descending toward 

54 

descending 

16 

descending away 

127 

Totals 

106 

99 

35 

240 


Chi-square ■ 2.760, p. < 0.50 ns 


The tendency to turn toward the intruding aircraft may be expressed as a pair 
of counts for each pilot. The first count is the total number of maneuvers classi- 
fied as turns toward, that is, the sura of climbing-turns toward, level-turns toward, 
and descending-turns toward. The second count is the total number analogously clas- 
sified as turns away. One may use these pairs to test both the direction and magni- 
tude of the preference for each pilot. Since previous experiments have suggested 
the existence of a bias to turn toward the intruding aircraft (Palmer et al., 1981), 
a one-tailed test for a similar bias may be justified. The tendency to turn toward 
the intruding aircraft occurs in seven of ten pilots; the two remaining pilots showed 
no turning preference. A sign test across subjects on this preference show it to be 
marginally reliable (sign test; p. < 0.035, one-tailed, but p. < 0.07, two-tailed). 
The withln-subjeot sign tests show that only five of the ten subjects exhibited reli- 
able biases to turn toward the Intruder. Therefore, although this within-sub j ect 
analysis may not be very powerful due to the relatively small number of maneuvers 
analyzed, the bias may not be representative of the entire pilot population. Future 
experiments will allow an Increase in the size of the sample of pilots and the number 
of maneuvers, and should allow a more statistically powerful assessment of the bias. 
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TABLE 4.- MANEUVER PREFERENCES FOR EACH SUBJECT 


Subjects 

Turns 

toward 

Turns 

away 

Climbs 

Descents 

Maneuvers 
keeping 
intruder 
in sight 

Maneuvers 
not keeping 
intruder 
in sight 

1 

21 

€ 

0 

3 

24 

0° 

2 

8 


0 

2 

10 

0° 

3 

21 

i« 

0 

22° 

21 

1° 

4 

11 

11 

5 

14 

12 

12 

5 

14 

0“ 

11 


11 

11 

6 

16 

qO 

19 

0° 

3 

19 

7 

6 

11 

8 

8 

10 

14 

8 

0 

0 

10 

14 

14 

10 

9 

12 

7 

0 

5 

17 

7 

10 

2 

0 

0 

24° 

24 

0° 


?p. < 0,05 
»p, < 0.025 
< 0.01 


A similar method can be used to detemtine each subject's preference of vertical 
maneuvers, but for this case a one-tailed test may not be justified by previous 
results. Of the nine subjects exhibiting a preferred direction of vertical maneuver, 
seven show a preference for descf'uding maneuvers; only one subject chose an equal 
number of ascending and descendiivj maneuvers. Thus, the tendency to descend does not 
stand up as reliably across subjects as the tendency to turn toward the intruding 
aircraft (sign tests p. <0.09, one-tailed, p. < 0.18, two-tailed). 

Since discussions with the pilots after the experiment isidicated that their 
choice of maneuver may have been motivated by the desire to keep the intruder poten- 
tially "in sight," maneuvers for each pilot were classified as (1) those that would 
keep the intruder "in sight" and (2) those that would not. Because all intruders 
were at ownship's altitude, any maneuver incorporating either a climb or turn away 
could be considered one that would not keep it "in sight." Thus, only descents, 
turns toward, and descents toward would keep the Intruder "in sight." Since there 
are three cells for "in sight" maneuvers and five cells for "not in sight" maneuvers 
and the null hypothesis assumes equal number of msT.-euvers per cell, a bias to 
maneuver so as to keep the intruder in sight would manifest Itself by a ratio of 
"in sight"/"not in sight" exceeding 3/5, Nine of the ten subjects show such a ratio 
(binomial tests p, < 0.028, two-tailed). However, this result should be tempered 
by the observation that within-subject chi-square tests of each subject's "in sight"/ 
"not in sight" ratios show only four subjects with reliable biases. 


DISCUSSION 


The dominance of time to minimum separation as a factor influencing "subjective 
threat" is an Interesting finding since the related measure, tau or range/range rate, 
is an important factor Included in automatic alarm systems (Lenz et al. , 1980), The 
pilot's choices of maneuvers, however, are not congruent with the avoidance maneuvers 
typically recommended by these logics. The automatic systems often chose maneuvers 
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that will tnaxlml^Q minimum separation, which in the horizontal case usually involves 
a turn away rather than a turn toward an Intruder (Palmer, Jago, and DuBord, 1981). 

The choice of the turns toward Intruders, as well as the descents, may have been 
caused by the pilots' disposition to maneuver so as to keep the Intruder potentially 
in sight. Several of the pilots mentioned this fact spontaneoijsly in their debrief- 
ing. In fact, the description of their maneuvers os turns toword the intruding air- 
craft is not completely general, since they were actually attempting to turn so os 
to go behind the intruder, a maneuver that has the effect of minimizing time to 
resolve the traffic conflict. Nevertheless, their choice is especially Interesting 
since all subjects were told to assume IMC conditions and that they therefore could 
not expect to be able to see the intruder out the window.^ 

A number of display conditions used for this experiment may have affected the 
results in specific ways. The choice of a 30 n. mi. map range, for example, may 
have made the display of the miss distances of 0.5 and 1.0 n. ml. appear as rela- 
tively indiscriminable distances. This display condition could have reduced the 
Influence of minimum miss distance in determining subjective threat. Similarly, the 
choice of a scenario in which the pilot was cleared for an approach to Denver may 
have influenced Che use of vertical evasive maneuvers: the pilots may have had the 

presumption that airspace below them was available. 

The choice of the ranges of each of the experimental variables was, however, 
made to cover a "range of concern" in an attempt to examine situations where the 
"subjeetlye threat judgment" of the pilots might be an important issue in the resolu- 
tion of a dangerous conflict. Thus, attempt to maximize the power of any examina- 
tion of encounter parameter was temporized by an attempt to keep the display situa- 
tions realistic and relevant toward the use of CDTI in actual traffic situations. 

Subsequent experiments will investigate some of the biases and effects discussed 
here as well as their interactions. Specific factors to be investigated will be the 
effect on subjective threat of the choice of different map ranges, the choice of a 
departing rather than arriving scenario, the effect of traffic at different altitudes 
with either closing or diverging vertical speed, and the Influence of the predictor 
types. 
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®Note that the resistance to training of this maneuvering is not addressed by 
this experiment. 


10 


APPENDIX 


COCKPIT DISPUV OP TRAFFIC INFORMATION 
INTRODUCTION 


The photographs of Cockpit Display of Traffic Information (CDTI) present 
encounters with intruding aircraft of various levels of threat. The purpose of the 
experiment is to assess the subjective dimensions of threat. Encounters prssented 
vary in several objective dimensions such as miss distance, time to minimum miss 
distance, relative velocity, and approach angle. 

These parameters enter into the decision logic of potential collision avoidance 
systems and we wish to determine if these systems make decisions consistent with the 
pilots' perceived level of threat. The specific encounters we will present are 
somewhat arbitrary and you should consider the Intruder a "Maverick" who is not part 
of an approach pattern and who will continue his current flightpath, 


SYMBOLOGIES 


OWNSHIP; 


NOTE', 



32 sec flightpath predictor 


If ownship were in a 3 “/sec turn its predictor would look like this! 



INTRUDER! 


A 


UFO 


32 sec flightpath predictor 

32 sec flightpath history 

•data tag 


XXX-YYY 


XXX — Barometric Altitude (ALT), measured in hundreds of feet (i.e,, 100 corre- 
sponds to 10,000) 

YYY — True Air Speed (TAS) , measured in knots per hour 

NOTE: Actual positions of the aircraft are at the tip of the symbols that rep- 

resent them. 
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DISPLAY CONSOLE; 



MR 30 miles 


NOTE: Assume intruder continues at current heading, speed, and altitude for the 

individual encounters. 

NOTE: Ownship speed and altitude are indicated as being the same in all photo- 

graphs; these as well as heading could be changed. 

SOME ADDITIONAL ASSUMPTIONS: 

Assume IMG conditions (i.e., you can't visually see other aircraft). 

Assume intruder has a transponder but no CAS or CDTI system (i.e., intruder 
doesn't see you). 

Assume cleared for normal approach to 26L at Denver Stapleton Airport. 


MR (Map Range) — Measured from ownship to top of display 
HDG (Heading) — Heading indicator, no wind: track UP 

Navigation points: 

MM, OM — Middle/Guter Marker 
WI — Way point 

IOC - VOR/DME 
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INSTRUCTIONS - EXPERIMENT A 


With the given photographs of 25 possible encounters, evaluate and sort all 25 
as to the degree of collision threat that you feel is caused by the Intruder air- 
craft. Order the photos in a stack using the following scale as a guideline; 

Arranged order Degree of threat 

Top of stack Critically high 
Very high 
High 
Mediuw 

Bottom of stock Low 
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INSTRUCTIONS - EXPERIMENT B 


This experiment has two parts for each photo; 

1) Rate the degree of action. 

2) Select a maneuver you would be inclined to use. 


PART 1: 

The given 25 displays of possible encounters have varying degrees of threat of 
collision. Evaluate each of the 25 displays according to degree of action you feel 
would need to be taken to avoid collision. Rate the displays Individually by select- 
ing a degree of action number (DA //) between 0.0 and 10.0 that you feel represents 
the course of action needed. Use the following scale as a guideline: 

DA // Degree of action 

0.0 Ignore 

2.0 Continue monitoring 

4.C Possible evasive (maneuver likely within the next 30 sec) 

6.0 Probable evasive (maneuver likely within the next 15 see) 

8.0 Immediate evasive (maneuver right now) 

10.0 Violent evasive (should have maneuvered sooner) 

NOTE: You may select any DA // (0.0-10.0) to represent the degree you feel is 

necessary (i.e., 5.0, 5.4, 5.6, 7.0, 7.2, etc.). 


PART 2: 

As you also rate the encounters individually, select a maneuver for each that 
you feel you might do in order to decrease the threat of collision. Use the figure 
below as a guide for the maneuvers you would make: 


Climb 

Climb 

Climb 

Left 


Right 

Left 

None 

Right 

Descend 


Descend 

Left 

Descend 

Right 
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